User terminal device with pen and controlling method thereof

ABSTRACT

A user terminal device capable of attaching and detaching a pen is provided. The device includes a detector configured to detect a user manipulation regarding a screen, and a controller configured to change a layout of the screen to correspond to a pen use mode in response to the pen being removed from the user terminal device, perform a control operation corresponding to the detected user manipulation in response to the user manipulation using the pen being detected, and restore the layout of the screen to its original state in response to the pen being remounted on the user terminal apparatus.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Korean Patent Application Nos.10-2013-0046995, filed in the Korean Intellectual Property Office onApr. 26, 2013, and 10-2013-0097737, filed in the Korean IntellectualProperty Office on Aug. 19, 2013, the disclosures of which areincorporated herein by reference.

BACKGROUND

1. Field

Aspects of the exemplary embodiments relate to a user terminal deviceand a controlling method thereof. In particular, exemplary embodimentsrelate to a user terminal device capable of performing various controloperations using a pen and a controlling method thereof.

2. Description of the Related Art

With the development of electronic technologies, various types ofelectronic apparatuses have been developed and distributed. Inparticular, a lot of portable user terminal devices such as a mobilephone, a tablet PC, a laptop PC, etc., have been widely developed anddistributed.

These user terminal devices may provide a plurality of services usingvarious applications. A user may execute a desired application byselecting an icon displayed on a home screen or an application iconscreen.

The home screen refers to a screen which is displayed for the first timewhen the user terminal device is turned on in a usable state.

A user may set the home screen such that icons corresponding to variousdesirable functions, content icons, widgets, etc., may be displayed onthe home screen. The user may perform control operations correspondingto each icon by touching the icons displayed on the screen using his orher finger.

A related art user terminal device, which is capable of performingcontrol operations using a plurality of inputters such as a pen insteadof a finger, has been distributed. However, in the related art userterminal device, no difference can be found between a finger and a penwhen it comes to performing control operations.

Thus, a technology is required for allowing a user to perform controloperations using a pen more conveniently and effectively.

SUMMARY

An aspect of the exemplary embodiments relates to a user terminal devicewhich provides a user interaction for performing control operationsusing a pen more conveniently and swiftly, and a controlling methodthereof.

According to an aspect of an exemplary embodiment, a user terminaldevice capable of attaching and detaching a pen and performing a userinput through a plurality of inputters includes a display configured todisplay a screen including a plurality of objects, a detector configuredto detect a user manipulation through the inputters, and a controllerconfigured to arrange and display a plurality of objects on a screenaccording to a first layout in response to a first inputter being used,among the plurality of inputters, and to re-arrange and display theplurality of objects on the screen, the objects being previouslyarranged according to the first layout, according to a second layout inresponse to a second inputter being used, among the plurality ofinputters.

The second inputter may be a pen, and the controller may be configuredto detect whether the pen is removed from the user terminal device andarrange the plurality of objects according to the second layout.

The first layout and the second layout may be different from each otherin terms of display attributes of the plurality of objects.

The screen may include a menu area configured to display at least onemenu which is selectable by the pen and a content display areaconfigured to display at least one content, and the controller mayreduce a size of the menu area and enlarge a size of the content displayarea to display the screen according to the second layout in response tothe pen being removed from the user terminal device, and the controlleris configured to restore the size of the menu area and the size of thecontent display area to the screen according to the first layout inresponse to the pen being mounted in the user terminal device.

The controller may be configured to reduce a size of each object of theobjects, and increase a number of total objects displayed on the screen.

The screen may include an indicator area configured to display at leastone indicator, and the controller, may perform a control operationcorresponding to a selected indicator in response to one of the at leastone indicator displayed on the indicator area being selected by the pen.

The controller may be configured to change a shape of a selectedindicator to a shape corresponding to another mode, and convert acurrent mode of the user terminal device to the another mode in responseto an indicator corresponding to a current mode of the user terminalbeing selected by the pen.

The controller may be configured to display a first quick panel screenin response to a general touch dragging manipulation being detected froman edge area of the screen towards a center of the screen, and thecontroller may be configured to display a second quick panel screen inresponse to pen dragging manipulation being detected from the edge areaof the screen towards the center area of the screen.

The controller may be configured to perform a control operation which ismatched with a tap according to a predetermined pattern of a detectedpen manipulation in response to a pen manipulation in the predeterminedpattern being detected on the tap displayed on the second quick panelscreen.

The first quick panel screen and the second quick panel screen may beconfigured to display at least one respective notification item, and thecontroller may be configured to perform a control operationcorresponding to a notification item and a note-taking automatically inresponse to the note-taking being performed using the pen on thenotification item displayed on the second quick panel screen.

According to another aspect of an exemplary embodiment, a method forcontrolling a user terminal device includes, displaying a screen whichincludes a plurality of objects arranged according to a first layout inresponse to a pen being mounted on the user terminal device, changingthe screen by re-arranging the plurality of objects on the screen, theobjects being previously arranged according to the first layout,according to a second layout in response to the pen being removed fromthe user terminal device, and re-arranging the plurality of objects onthe screen according to the first layout in response to the pen beingmounted again on the user terminal device.

The screen may include a menu area configure to display at least onemenu which is selectable by the pen and a content display areaconfigured to display at least one content, and the changing the screenmay include reducing a size of the menu area and enlarging a size of thecontent display area in response to the pen being removed from the userterminal device.

The changing the screen may include reducing a size of each object ofthe objects, and increasing a number of total objects displayed on thescreen in response to the pen being removed from the user terminaldevice.

The screen may include an indicator area configured to display at leastone indicator, and the method may further include performing a controloperation corresponding to a selected indicator in response to one ofthe at least one indicator displayed on the indicator area beingselected by the pen.

The method may further include changing a shape of a selected indicatorto a shape corresponding to another mode, and converting a current modeof the user terminal device to the another mode in response to anindicator corresponding to the current mode of the user terminal devicebeing selected by the pen.

The method may further include displaying a first quick panel screen inresponse to a general touch dragging manipulation being detected from anedge area of the screen towards a center area of the screen, anddisplaying a second quick panel screen in response to a pen manipulationof dragging being detected from the edge area of the screen towards thecenter area of the screen.

The method may further include performing a control operation which ismatched with a tap according to a predetermined pattern of a detectedpen manipulation in response to a pen manipulation in the predeterminedpattern being detected on the tap displayed on the second quick panelscreen.

The method may further include performing a control operationcorresponding to a notification item and a note-taking automatically inresponse to the note-taking being performed using the pen on thenotification item displayed on the second quick panel screen.

According to another aspect of an exemplary embodiment, a user terminaldevice capable of attaching and detaching a pen includes a displayconfigured to display a screen including an indicator area composed of aplurality of indicators corresponding to an operation state of the userterminal device, a detector configured to detect a user manipulation,and a controller configured to operate in a general mode in response tothe pen being mounted on the user terminal device, and to operate in apen use mode in response to the pen being removed from the user terminaldevice, and the controller is further configured to disregard a touch inresponse to the touch being input in the general mode to select aplurality of indicators, and to perform a control operationcorresponding to a touched indicator in response to one of the pluralityof indicators being touched by the pen in the pen use mode.

The indicator area may include at least one of a first indicator tonotify a missed call, a second indicator to notify a newly-receivedmessage or a messenger, a third indicator to notify a program update, afourth indicator to notify a current communication method, a fifthindicator to notify a communication state, a sixth indicator to notify aresidual battery, a seventh indicator to notify whether an alarm is set,an eighth indicator to notify time, a ninth indicator to notify date, atenth indicator to notify weather information, an eleventh indicator tonotify that a predetermined schedule has come, a twelfth indicator tonotify an operation mode of the user terminal device, a thirteenthindicator to notify whether it is a vibration mode, and a fourteenthindicator to notify whether Bluetooth is used.

According to another aspect of an exemplary embodiment, a user terminaldevice capable of attaching and detaching a pen includes a displayconfigured to display a home screen which divided into a plurality ofpages and is convertible by page unit, a detector configured to detect auser manipulation, and a controller configured to convert and displaythe home screen by the page unit according to the user manipulation inresponse to the pen being mounted on the user terminal device, and toreduce each size of the plurality of pages and display entire pageswithin a single home screen, and at least one object is displayed oneach page of the home screen.

The controller may be configured to move the object to another pageaccording to a pen manipulation in response to the pen manipulation formoving the object included in one page to the another page beingdetected while the entire pages are displayed on the single home screen.

According to another exemplary embodiment, a user terminal devicecapable of attaching and detaching a pen includes a display configuredto display a screen including a widget, a detector configured to detecta user manipulation, and a controller configured to operate in a generalmode in response to the pen being mounted on the user terminal device,and to operate in a pen use mode in response to the pen being removedfrom the user terminal device, and the controller is configured toexecute a program corresponding to the widget in response to a touchbeing input on the widget in the general mode, and to adjust settingdata of the widget according to a pen manipulation and change adisplayed content of the widget according to the adjusted setting datain response to the pen manipulation regarding the widget being input inthe pen use mode.

The widget may be a widget in a form of an analog clock including a hourhand and a minute hand, and the controller may be configured to set analarm based on a time changed according to a pen manipulation inresponse to the pen manipulation being input for moving at least one ofthe hour hand and the minute hand on the analog clock, and move at leastone of the hour hand and the minute hand to a time when the alarm isset.

According to an aspect of an exemplary embodiment, a user terminaldevice capable of attaching and detaching a pen includes a displayconfigured to display a screen including a plurality of objects, adetector configured to detect a user manipulation regarding the screen,and a controller configured to arrange the plurality of objects on thescreen according to a first layout in response to the pen being mountedon the user terminal device, and to change the screen by rearranging theplurality of objects according to a second layout corresponding to a penuse mode in response to the pen being removed from the user terminaldevice, and the controller is configured to perform a control operationcorresponding to a selected object in response to at least one of theplurality of objects being selected by the pen while the screen, whichis changed according to the second layout, is being displayed, and thecontroller is configured to rearrange the plurality of objects accordingto the first layout in response to the pen being re-mounted on the userterminal device.

According to yet another exemplary embodiment, a method of controlling auser terminal device includes displaying a screen, displaying a firstquick panel on the screen for setting an operation or one of a pluralityof main functions of the user terminal device in response to determiningthat a first input is dragged from a border of the screen towards acenter of the screen, and displaying a second quick panel on the screenfor setting one of a plurality of minute control operations in responseto determining that a second input is dragged from the border of thescreen towards the center of the screen. The first input is differentfrom the second input.

According to the above-described various exemplary embodiments, a userinteraction method which is appropriate for using a pen may be provided.Accordingly, the function of the user terminal device may be controlledmore easily and conveniently.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects of the exemplary embodiments will be moreapparent by describing certain exemplary embodiments t with reference tothe accompanying drawings, in which:

FIG. 1 is a view illustrating external configuration of a user terminaldevice according to an exemplary embodiment;

FIG. 2 is a block diagram illustrating configuration of a user terminaldevice according to an exemplary embodiment;

FIG. 3 is a flowchart provided to explain a method for controlling auser terminal device according to an exemplary embodiment;

FIG. 4 is a block diagram illustrating configuration of a user terminaldevice capable of distinguishing a manipulation using a body of a userfrom a manipulation using a pen;

FIG. 5 is a view illustrating an example of configuration of a display;

FIGS. 6 and 7 are views provided to explain various methods fordetecting a proximal manipulation of a user;

FIGS. 8 and 9 are views illustrating an example of method for changing ascreen depending on whether a pen is removed or not;

FIGS. 10 to 17 are views provided to explain various interaction methodsusing a pen;

FIG. 18 is a flowchart provided to explain an exemplary embodiment forchanging a shape of a quick panel screen according to an operation mode;

FIG. 19 is a view provided to explain the exemplary embodiment of FIG.18;

FIGS. 20 to 25 are flowcharts provided to explain various examples of apen interaction operation on a quick panel screen;

FIG. 26 is a view illustrating an example of a home screen which can beset directly by a user;

FIG. 27 is a view provided to explain a method for providing differenthome screens depending on whether a pen is mounted or not;

FIGS. 28 to 30 are views provided to explain various pen interactionoperations on a widget;

FIG. 31 is a block diagram provided to explain comprehensiveconfiguration of a user terminal device according to various exemplaryembodiments; and

FIG. 32 is a view illustrating an example of configuration of softwarewhich can be used in a user terminal device according to an exemplaryembodiment.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

The method steps and system components have been represented by relatedart symbols in the figure, and only specific details which are relevantfor an understanding of the exemplary embodiments have been shown.Further, details which have not been disclosed may be readily apparentto a person of ordinary skilled in the art. In the exemplaryembodiments, relational terms such as first and second, etc., may beused to distinguish one entity from another entity, without necessarilyimplying any actual relationship or order between such entities.

FIG. 1 is a view illustrating an external configuration of a userterminal device according to an exemplary embodiment. According to FIG.1, a display 110 is disposed on the front of a user terminal device 100,and a pen fixing unit 190 is provided on one side. A pen 200 may bemounted in the pen fixing unit 190.

In FIG. 1, the pen fixing unit 190 is provided in the form of a holesuch that a pen 200 may be inserted therein. However, the pen fixingunit 190 may be realized in other forms. For example, the pen fixingunit 190 may be formed on the side of the user terminal device 100 so asto fix the pen 200 in the fixed clip form.

The user terminal device 100 refers to various types of electronicdevices which may be arbitrarily used by a user. In FIG. 1, the userterminal device 100 is illustrated as a mobile phone, but the types ofthe user terminal device 100 may be realized in many different ways. Tobe specific, the user terminal device 100 may be realized as varioustypes of devices such as mobile phone, tablet PC, laptop PC, PDA, MP3player, electronic album device, TV, PC, kiosk, etc.

FIG. 2 is a block diagram provided to explain configuration of a userterminal device according to an exemplary embodiment. According to FIG.2, the user terminal device 100 includes the display 110, a detector120, and a controller 130. In FIG. 2, only those elements which areinvolved in operations according to various exemplary embodiments areillustrated, and other detailed elements are omitted.

The display 110 may display various types of screens such as a homescreen, an icon screen, a list screen, an application execution screen,a web browser screen, a content play screen, etc. Various objects suchas an icon, an image, and a text may be displayed on each screen. A usermay select those objects.

The detector 120 may detect a user manipulation with respect to thedisplay 110. In particular, when a user touches one of the objectsdisplayed on the screen, the detector 120 may detect the touched pointand notify the controller 130.

The controller 130 may control overall operations of the user terminaldevice 100. When it is detected by the detector 120 that one icon isselected, the controller 130 performs a control operation correspondingto the one icon.

The controller 130 may determine a mode according to which inputters isused, among a plurality of inputters capable of manipulating the userterminal device 100, and may change a screen layout according to thedetermined inputters. The inputters may include a part of a user bodysuch as a finger, other electric conductors, the pen 200, etc. By usingsuch various inputters, a user may control the operations of the userterminal device 100.

In a first mode where a first inputter, among a plurality of inputtersis used, the controller 130 arranges and displays objects on the screenof the display 110 according to a first layout. Further, in a secondmode where a second inputter is used, the controller 130 rearranges theobjects which have been arranged according to the first layout accordingto a second layout, and displays the rearranged objects.

The first layout and the second layout represent layouts havingdifferent properties of displaying a plurality of objects. The displayproperties may include various properties such as size, form, color,location of display, criteria of arrangement, method of arrangement, thenumber of objects displayed on each line, etc. The second inputter maybe the pen 200, and the first inputter may be various electricconductors such as a user body part or other elements except for a pen.Subsequently, various display properties such as size, form, color, anddisplay location of an object displayed on the screen may vary dependingon the type of inputters.

The controller 130 may determine the type of inputters in various ways.For example, the controller 130 may determine an inputters to be useddepending on whether the pen 200 is attached to or detached from themain body of the user terminal device 100. In particular, when the penis mounted on the main body of the user terminal device 100, thecontroller 130 may determine that it is in the first mode, and when thepen is detached, the controller 130 may determine that it is in thesecond mode.

The pen may be attached or detached in various ways. For example, if thepen fixing unit 190 is realized in the form of hole as illustrated inFIG. 1, there may be a contacting point or a button inside the holewhich may be in contact with the pen 200. When the pen 200 is mounted onthe pen fixing unit 190, the pen 200 may be in contact with thecontacting point or the button may be pressed. In this case, a signal ofa certain size may be input to the controller 130 by an electric circuitconnected to the pen fixing unit 190. Further, when the pen 200 isdetached from the pen fixing unit 190, it is also detached from thecontacting point or the button. Accordingly, the level of the signalapplied to the controller 130 from the pen fixing unit 190 may change.Based on such change in the signal, the controller 130 may determinethat the pen 200 is detached.

The controller 130 may arrange and display a plurality of objects on thescreen according to the first layout while the pen is mounted. Further,when it is determined that the pen 200 is detached, the controller 130changes the layout of the screen displayed on the display 110 to thesecond layout. The second layout refers to a form of layoutcorresponding to a pen use mode. In other words, as the pen 200 issharper than a finger, it allows more accurate touch than a finger.Therefore, when using the pen 200, it is possible to select a menu ofsmaller size. In addition, when a coil is mounted inside the pen 200, itis possible to distinguish a hovering and a touch using the pen 200.

When at least one object is selected by the pen 200 while the objectsare rearranged according to the second layout, the controller 130performs a control operation corresponding to the selected object.

As such, the user terminal device 100 may realize different layoutsdepending on whether a pen is used or a finger is used. The differentlayouts may also include different contents on the screen. Examples ofthe first layout and the second layout will be described later withreference to the corresponding drawings.

After the pen 200 is detached from the user terminal device 100, theuser terminal device 100 is generally controlled by the pen. Thus, thecontroller 130 automatically enters into a pen use mode without anyseparate user manipulation. Further, when a pen is mounted on the userterminal device 100 again, the controller 130 returns to a general mode.Accordingly, the layout of the screen is also changed to the firstlayout, which is the mode before the pen is detached, and the objects onthe screen are rearranged according to the first layout.

FIG. 3 is a flowchart provided to explain a method for controlling auser terminal device according to an exemplary embodiment. According toFIG. 3, the display 110 displays a screen including a plurality ofobjects which are arranged according to the first layout correspondingto a general mode while the pen 200 is mounted on the user terminaldevice 100 (S310). The screen may be a home screen, or other types ofscreens such as an icon arrangement screen, an application executionscreen, a content play screen, a quick panel screen, etc. At this state,when the pen is detached (S320), the controller 130 changes the layoutof screen which is currently displayed (S330).

In other words, the controller 130 changes a screen by rearranging aplurality of objects which are currently displayed on the screenaccording to the second layout corresponding to the pen use mode. Inparticular, the controller 130 changes the screen to a shapecorresponding to the pen use mode by adjusting at least one of variousproperties such as size, shape, location, number of objects which arecurrently displayed on the screen. Therefore, the objects which havebeen displayed on the screen are rearranged or reconfigured at the samethe pen is detached, such that the screen can be in a format which issuitable for using the pen. In this state, if the pen is mounted again(S340), the controller 130 restores the screen to a screen correspondingto the first layout (S350).

The layout may be changed in various shapes and methods according to thetype of the screen. In other words, as described above, the display 110may display various screens such as a home screen, an icon screen, alist screen, an application execution screen, a web browser screen, acontent play screen, etc.

The home screen refers to a screen which is displayed for the first timeafter the user terminal device 100 is turned on and preparation forsystem is completed. The home screen may also be referred to as a mainscreen, a basic screen, an initial screen, etc. The home screen may be abasic home screen provided by the manufacturing company of the userterminal device as a default, or may be a user creative screen which iscreated as the user matches a control operation with an object bydirectly drawing an icon or other objects. The method for configuring auser creative screen and the method for using the screen will beexplained in detail later.

The icon screen refers to a screen where icons regarding the entireapplications installed in the display 110 are displayed, and the listscreen refers to a screen where information regarding the entireapplications is arranged and displayed in the form of list. Theapplication execution screen refers to a screen which is rendered by theexecution of application, and the web browser screen refers to a screenwhere a web page received from an external web server is displayed bythe execution of a web browser. The content play screen refers to screenwhere a multimedia content is reproduced and its video frame isdisplayed. In addition, the display 110 may display various types ofscreens.

As described above, the controller 130 operates in various modesdepending on whether a pen is attached or detached, thereby performinginteraction operations which are appropriate for each mode. For example,a screen according to the first layout may be displayed while a pen ismounted. In this state, if a user selects an object on the screen usinghis or her finger or other things, the controller 130 may execute aprogram corresponding to the object and display a corresponding screen.

On the other hand, the controller 130 may display a screen includingobjects which are rearranged according to the second layout while thepen is detached. In this state, a user may perform various interactionoperations using the pen 200 in addition to his or her finger. Thecontroller 130 performs a control operation corresponding to the userinteraction operation.

The user terminal device 100 may perform a control operation using notonly the pen 200, but also a body part such as the user finger. The userterminal device 100 may distinctively recognize a touch or an approachusing the user body part and a touch or an approach using the pen 200.An approach may also be referred to as hovering. Accordingly, differentcontrol operations may be performed depending on the type and method ofinputters.

The specific configuration of a user terminal device which maydistinctively recognize the type and input method of inputters will bedescribed.

FIG. 4 is a block diagram illustrating a detailed configuration of auser terminal device capable of distinguishing a pen manipulation.Referring to FIG. 4, the user terminal device 100 includes a display110, a detector 120, a controller 130, and a storage 140.

The storage 140 is an element to store various programs and data whichare necessary to operate the user terminal device 100. The controller130 controls overall operations of a user terminal device using variousprograms and data stored in the storage 140.

The controller 130 includes RAM 131, ROM 132, CPU 133, GraphicProcessing Unit (GPU) 134, and Bus 135. The RAM 131, the ROM 132, theCPU 133, the GPU 134, etc., may be connected through the bus 135.

The CPU 133 access the storage 140, and performs booting using anoperating system (O/S) stored in the storage 140. In addition, the CPU133 performs various operations using various programs, contents, data,etc., stored in the storage 140.

The ROM 132 stores a set of commands for system booting. If a turn-oncommand is input and power is supplied, the CPU 133 copies an O/S storedin the storage 140 in the RAM 131 according to the command stored in theROM 132 and executes the O/S to boot the system. If the booting iscompleted, the CPU 133 copies various application programs stored in thestorage 140 in the RAM 131 and executes various programs copied in theRAM 131 to perform various operations.

When the booting of the user terminal device 100 is completed, the GPU134 displays a home screen. Specifically, the GPU 134 may generate ahome screen including various objects such as an icon, an image, and atext using a computing unit (not shown) and a rendering unit (notshown). The computing unit computes property values such as coordinates,shape, size, and color of each object to be displayed according to thelayout of the screen. The rendering unit generates a screen with variouslayouts including objects based on the property values computed by thecomputing unit. The screen generated by the rendering unit is providedto the display 110 and displayed within a display area.

In addition, when various applications stored in the storage 140 areexecuted, the GPU 134 displays execution screens of the correspondingapplications using various layout information and menu informationincluded in the applications. In addition, when a command to display anicon screen is input, the GPU 134 displays the icon screen where iconsof various applications stored in the storage 140 are arranged.Therefore, the GPU 134 may display various types of screens.

Meanwhile, the GPU 134 may change the layout of each screen according toan operation mode of the CPU 133. For example, if a plurality of objectsare arranged according to the first layout in a general mode, the GPU134 may arrange and display the same objects according to the secondlayout in a pen use mode. In addition, the screen may be changed invarious forms according to whether a mode is the pen use mode or not.

The display 110 displays various screens as described above. The display110 may be realized as various types of displays such as Liquid CrystalDisplay (LCD), Organic Light Emitting Diodes (OLED) display, PlasmaDisplay Panel (PDP), etc. The display 110 may further include a drivingcircuit, a backlight unit, etc., which may be realized in the form ofa-si TFT, low temperature poly silicon (LTPS) TFT, organic TFT (OTFT),etc.

In FIG. 4, the display 110 and the detector 120 are realized as a singletouch screen.

When a user manipulation is detected through various inputters, thedetector 120 notifies the detection result to the controller 130. Forexample, when a user touches one point of a screen using a finger or apen, the detector 120 notifies the x and y coordinates of the touchedpoint to the controller 130. When a user moves the touched point whiletouching the point, the detector 120 notifies the changed coordinates ofthe touched point to the controller 130. The detector 120 may detect anapproach of a user in addition to a direct touch.

In particular, the detector 120 may include a pen recognition panel 121and a touch panel 122. The pen recognition panel 121 detects an approachinput or a touch input of the pen 200, and outputs an event signalaccording to the detection result. The pen recognition panel 121 may berealized according to an EMR method, and may detect a touch or anapproach input according to the change in the intensity ofelectromagnetic field by an approach or a touch of a pen. In particular,the pen recognition panel 121 may include an electromagnetic inductioncoil sensor (not shown) and electromagnetic signal processing circuitunit (not shown). The electromagnetic induction coil sensor has a gridstructure where a plurality of loop-coils are transposed. Theelectromagnetic signal processing circuit unit provides AC signalssequentially to each of the loop coils of the electromagnetic inductioncoil sensor, and transmits signals output from each loop coil to thecontroller 130.

In the case where the pen 200 is configured to include a resonancecircuit, if the pen 200 exists near the loop coil of the pen recognitionpanel 121, the magnetic field transmitted from the corresponding loopcoil generates a electric current in the resonance circuit of the pen200 based on mutual electromagnetic induction. An induction magneticfield is generated from a coil which constitutes the resonance circuitof the pen based on the electric current, and the pen recognition panel121 detects the induction magnetic field from a loop coil which is inthe signal reception state, thereby detecting an approach location or atouch location of the pen. In the above exemplary embodiment, the pen200 is recited as a plurality of inputters, but any object capable ofgenerating an electric current based on magnetic induction may be usedas an inputters. The pen recognition panel 121 is disposed at the lowerportion of the display 110, and may be activated as a specific eventoccurs or as a default.

The touch panel 122 is an element to detect a physical touch input by auser body part or other objects. For example, the touch panel 122 may beprovided in the form of touch film, touch sheet, touch pad, etc. When atouch is detected, the touch panel 122 outputs a touch event valuecorresponding to the touched point.

As described above, the touch panel 122 may include various types ofsensors such as a capacitance-type sensor and resistive-type sensor. Forexample, when the touch panel 122 is realized as a capacitance-typesensor, the touch panel 122 may be coated on both sides of the glasseswith a thin metal conductive material (such as Indium Tin Oxide (ITO)layer) so that an electric current may flow on the surface of theglasses. Further, the touch panel 122 is coated and configured as adielectric substance which may store an electric charge. When thesurface of the touch panel 122 is touched, a certain amount of electriccharge is moved to the touched location by static electricity, and thetouch panel 122 detects the touched location by recognizing the changein the amount of electric current as a result of the movement of theelectric charge, and traces a touch event.

In FIG. 4, it is described that the pen recognition panel 121 and thetouch panel 122 are formed on separate panels from each other, but thetwo panels may be formed on a single panel. Accordingly, the detector120 may distinguish a touch manipulation and an approach manipulation bya user finger from a touch manipulation and an approach manipulation bya pen. The controller 130 may perform various control operationsaccording to the detection result.

The pen recognition panel 121 may recognize a manipulation by a penusing at least one pixel unit. Accordingly, even if the controller 130reduces the size of a menu displayed on the screen in a pen use mode, auser may easily select the corresponding menu. In order to detect aminute manipulation by a pen, the pen recognition panel 121 may beproduced to have a size which is greater than a display panel of thedisplay 110. Accordingly, a user manipulation may be detected using atleast one pixel unit from the border area of the display panel.

FIG. 5 is a view illustrating an example of detailed configuration ofthe display 110. According to FIG. 5, the display 110 may furtherinclude a processing circuit unit such as a timing controller 111, agate driver 112, a data driver 113, and a voltage driver 114, inaddition to a display panel 115. Although not illustrated in FIG. 5, thedisplay 110 may further include a frame buffer.

The timing controller 111 receives an external clock signal which issuitable for the display panel 115, a horizontal sync signal (Hsync), avertical sync signal (Vsync), etc., generates a gate control signal(scanning control signal) and a data control signal (data signal), andoutputs the signals to the gate driver 112 and the data driver 113,respectively.

The voltage driver 114 is an element to transmit a driving voltage tothe gate driver 112, the data driver 113, the display panel 115, etc.Specifically, VDD or VSS may be provided, which are necessary for thedisplay panel 115. In addition, a gate-on-voltage (Vgh) may be generatedand provided to the gate driver 112.

The gate driver 112 is connected to the display panel 115 throughscanning lines (S1, S2, S3, . . . , Sn). The gate driver 112 appliesgate on/off voltages (Vgh/Vgl) provided by the voltage driver 115 to thedisplay panel 115, according to a gate control signal generated by thetiming controller 111.

A data driver 233 is connected to the display panel 115 through datalines (D1, D2, D3, . . . , Dm). The data driver 233 inputs RGB data ofan image frame to the display panel 115 according to a data controlsignal generated by the timing controller 231.

The display panel 115 is configured such that a plurality of gate lines(GL1˜GLn) and a plurality of data lines (DL1˜DLn) are transposed to forman pixel area 116. If the display panel 115 operates according to anOLED method, each of the pixel area 116 may be configured to form lightemitting elements of R, G and B of an OLED. A switching element, i.e.,TFT, is formed on each pixel area 116. In addition, detection sensorsfor detecting a user manipulation using a body part or a pen may bedisposed at a lower part of the each pixel area 116. In other words, thedetector 120 may be disposed along with the display 110 so as to detecta user manipulation by at least one pixel unit.

As described above, a user may input a control command to the userterminal device 100 only through the operation of approaching aninputters, such as a finger or a pen, without directly touching thescreen of the display 110. In this case, the detector 120 may sense anapproached touch using a proximity sensor. The approached touch meansthat if a movement is recognized within a certain valid recognitionrange of a space while there is no direct touch on the screen, themovement is acknowledged as a touch gesture.

Referring to FIG. 6, the detector 120 may further include an infraredrays source unit 123 and an infrared rays sensor 124. The infrared rayssource unit 123 is disposed on one surface of the display panel 115, andthe infrared rays sensor 124 is disposed on the other surface of thedisplay panel 115. The infrared rays source unit 123 radiates infraredrays in a surface direction of the display panel 115. Accordingly, acertain area where an approach of a user's finger 50 may be recognizedexists on the surface which becomes a valid recognition area 5 where anapproached touch can be recognized.

The infrared rays sensor 124 may include a plurality of infrared rayssensing elements which are disposed in an array form. Accordingly, whenthe finger 50 approaches the valid recognition area 5, the infrared rayssensor 124 may detect infrared rays reflected by the finger 50 andgenerate an infrared rays scanning image. The user terminal device 100may detect an input of an approached touch using the generated infraredrays scanning image.

The detector 120 may also detect a case where the pen 200 approaches. Inother words, as described above, the pen recognition panel 121 mayinclude an electromagnetic induction coil sensor and an electromagneticsignal processing circuit unit (not shown).

FIG. 7 is a view illustrating an example of a configuration to detect anapproach of the pen 200 using the pen recognition panel 121. Accordingto FIG. 7, an electromagnetic induction coil sensor 125 may be furtherincluded in the display 110. When the pen 200 approaches, the magneticfield of the approached area changes by the magnetic field coil of thepen 200. Thus, a change in the magnetic field of the electromagneticinduction coil sensor 125 is detected. The detected magnetic field isprocessed by the electromagnetic signal processing circuit and input tothe controller 130. Based on the input value, the point approached bythe pen 200 can be detected. In addition, the controller 130 maydetermine whether there is a direct touch or not according the magnitudeof change in the value of the magnetic field.

As illustrated in FIG. 7, when the infrared rays source unit 123, theinfrared rays sensor 124, and the pen recognition panel 121 are used, itis possible to recognize whether a touch or an approach is made by auser body part such as the finger 50 or by the pen 200. The controller130 may perform various control operations selectively according to thetype and manipulation method of the inputters. As described above, thescreen may be provided in various ways and with various contents. Inaddition, as a user may manipulate the screen in various ways usingvarious inputters, the layouts may be changed according to the types ofthe screen. Hereinafter, an example of a screen change when a pen isattached and detached will be explained in detail.

<Change of Layout in Home Screen>

FIG. 8 is a view provided to explain an example of a controlling methodwhen a pen is detached while a home screen is displayed. The home screenconsists of a background screen 500 including various objects 510 suchas a widget and an icon.

In FIG. 8, it is illustrated that, at the time when a pen is mounted(A), each object 510 has the same size and shape in a matrix form.However, the size and shape of the objects 510 may be realized invarious ways. For example, if a widget is included, the widget may bedisplayed in the size and shape which are different from those of ageneral icon.

In FIG. 8, the first layout means a 3*3 matrix shape, and the secondlayout means a 5*7 matrix shape. In other words, the controller 130arranges and displays a plurality of objects 510 in the 3*3 matrix shape(i.e., the first layout) on the screen 500 while a pen is mounted (A).

In this state, if the pen is detached (B), the controller 130 reducesthe size of each object 510 so that the sizes of objects become smallerthan those when the pen is mounted. As the sizes of the objects 510become smaller, more objects 510 can be displayed on one screen 500.Therefore, according to FIG. 8, the controller 130 may arrange theobjects 510 in the 5*7 matrix shape on the screen 500 according to thesecond layout. As described above, when the pen 200 is detached, thelayout of the screen may be changed in consideration of the performanceof the pen 200 so that a user may use the user terminal device moreconveniently.

In FIG. 8, it is illustrated that only the size and the number ofobjects are changed, but the shape and other display attributes ofobjects may also be changed. For example, each object may have a squareshape in a general mode (i.e., when the pen is mounted) and the shapemay be changed in various ways in a pen use mode (i.e., when the pen isdetached) in order to induce a pen manipulation. For example, the shapeof the objects may be changed to a flag flying in the wind, a dartplate, an arrow target, etc. Alternatively, various animation effect maybe applied to each object.

In FIG. 8, the home screen 500 including a plurality of objects 510 isdescribed as an example, but the same exemplary embodiment can berealized in other screens, such as various application screens and iconscreens where various objects are arranged and displayed. For example,if a pen is detached while an icon screen where icons of the entireapplications installed in the user terminal device 100 are arranged isdisplayed, the controller 130 may reduce the size of each icon so thatthe number of icons displayed on the screen may increase. Alternatively,if a gallery program for displaying stored photo contents or videocontents is executed, the controller 130 may display a photo galleryscreen including a plurality of thumbnail images regarding each of thephoto contents and the video contents. If the pen is detached while sucha photo gallery screen is displayed, the controller 130 may reduce thesize of each thumbnail image while increasing the number of images so asto provide more thumbnail images on the screen. In addition, theexemplary embodiment of FIG. 8 may also be realized in other variousscreens such as a web browser screen.

FIG. 9 is a view provided to explain another example of the controllingmethod when the pen is detached while the home screen is displayed.According to FIG. 9, the display 110 may display a home screen 600 whichincludes a plurality of objects 610 such as a widget, an icon, etc.

The controller 130 displays each object 610 in a size which can betouched by a finger while the pen 200 is mounted (A). The home screen600 may be divided by a plurality of pages, and may be converted by apage unit. While the pen 200 is mounted, a user may touch the homescreen 600 which is currently displayed and drag or flick the screen inone direction so as to turn pages. When a page is turned, the display110 displays objects on the turned page.

When the pen is detached (B), the controller 130 may display the entirepages 600-1 through 600-4 on one screen 600. The objects 610 disposed bya user are displayed on each page 600-1 through 600-4. According to anexemplary embodiment, a user may check objects 610 displayed on theentire pages 600-1 through 600-4 only with the operation of detachingthe pen 200. In addition, a user may directly select the objects on eachpage 600-1 through 600-4 using the pen 200. When the object 610 isselected, the controller 130 performs a control operation which ismatched with the object 610. For example, if an object which is matchedwith a web browser is selected, the web browser screen may be displayed.

As described above, the exemplary embodiment of FIG. 9 may also beapplied to other types of screens in addition to the home screen.

As described above, when the screen is rearranged according to thesecond layout in the pen detachment mode, a user may perform variousinteraction operations by touching or hovering each object on the screenusing the pen 200 in addition to a finger. In particular, as the pen 200is capable of performing a minute touch as described above, it ispossible to perform various control operations using the pen rather thana finger. The examples of various control operations which can beperformed by a pen while the pen is detached and the layout of thescreen is changed will be described in detail.

According to another exemplary embodiment, the controller 130 performsvarious control operations according to a pen manipulation while the penis detached and the objects on the screen are rearranged and displayed.For example, when there is a hovering, a first control operationcorresponding to the hovering is performed. Further, when there is atouch, a second control operation corresponding to the touch isperformed. The first control operation and the second control operationare different types of operations from each other.

FIG. 10 is a view illustrating an example of a control operation whichis performed using a pen. As described above with reference to FIG. 8,when the pen is detached, the controller 130 may reduce the size of eachobject 1210 so that more objects can be displayed on one screen 1200.The size of the object 1210 may be reduced to an extent where the objectcan be identified by human eyes. For example, in the case of an objectof which size is 1 cm in length and width in a general mode, the sizecan be reduced less than 0.5 cm in the pen use mode. However, this isonly an example, and the size of an object is not limited thereto.

In this case, when a user performs a hovering manipulation ofapproaching the pen 200 to one object 1210, the controller 130 mayenlarge the size of the object 1210 so that the user may view thecontent of the corresponding object 1210 more easily. The user maydetermine whether to touch each object 1210 through a hoveringmanipulation while viewing the object 1210 more accurately through hisor her eyes. If the user touches the corresponding object 1210, thecontroller 130 performs a control operation matched with the object1210. In other words, if a hovering is detected in FIG. 10, thecontroller 130 performs the first control operation of enlarging theobject. When a touch is detected, the controller 130 performs the secondcontrol operation of executing an application matched with thecorresponding object. The size of the object 1210, which is enlargedwhen a hovering is detected, may be set as a default or may bedetermined by a user. The size may vary depending the distance ofhovering.

FIG. 11 is a view illustrating another example of a control operationwhich is performed using a pen. According to FIG. 11, when the pen 200is hovering on one icon 1310 while a screen 1300 including a pluralityof icons 1 through 9 is displayed, the controller 130 displays a pop-upscreen 1320 including various information regarding the correspondingicon 1310 on one side. The pop-up screen 1320 may display at least oneof a title of the icon, a generation date of the icon, the capacity ofan application matched with the icon, a memory occupancy rate, etc. Inaddition, if a number notification is displayed on one side of the icon1310, the detailed description regarding the number notification may bedisplayed when the pen is hovering.

FIG. 12 is a view illustrating another example of a control operationwhich is performed using a pen. According to FIG. 12, if a hovering ofthe pen 200 with respect to a folder icon 1410 is detected while ascreen 1400 including the folder icon 1410 is displayed, the controller130 enlarges the folder icon 1410 and displays the entire lists 1411through 1414 regarding contents of the folder such as icon, photo, etc.,on the folder icon 1410. A user may directly select various icons,photos, and other contents displayed on the folder icon 1410 using thepen 200. The controller directly performs a control operationcorresponding to the selected object. For example, if a photo isselected, the controller 130 may enlarge the photo and display the photoon the full screen. According to such an exemplary embodiment, a usermay check and select the contents of each folder without even opening upeach of the folders.

FIG. 13 is a view illustrating another example of a control operationwhich is performed using a pen. As described above with reference toFIG. 9, if the pen is detached (B) while the screen 600 of one page,among a plurality of pages is displayed (A), the controller 130 maydisplay the entire pages 600-1 through 600-4 on one screen 600.

In this case, a user may move the location of the object 610 betweeneach of the pages 600-1 through 600-4 using the pen 200. FIG. 13illustrates the process of touching one object (e.g., 9) on the firstpage 600-1 and dragging it to the third page 600-3. Accordingly, iconscan be moved between pages easily.

As described above, the controller 130 may change the layout of the homescreen depending on whether the pen is mounted or detached while thehome screen is displayed. In some cases, not only the layout of thescreen but also the launcher of the home screen may be changed. Theoperation of displaying different layouts depending on whether the penis attached or detached may also be applied to other screens than thehome screen, such as icon screen, application execution screen, quickpanel screen, etc. An example of a change of layout in an applicationscreen as a result of execution of the application will be described indetail.

<Change of Layout in Application Screen>

FIG. 14 is a view provided to explain an example of controlling methodwhen a pen is detached. According to FIG. 14, a screen 400 may bedivided into a menu area 410 for displaying at least one menu which isselectable by a user and a content display area 420 for displaying atleast one content. In particular, when a program (e.g., a galleryprogram) for viewing multi-media contents such as photos and videosstored in the user terminal device 100 is executed, i.e., a program toreproduce music contents or video contents, various game programs, etc.,are executed, the controller 130 may display the screen 400 asillustrated in FIG. 14.

According to FIG. 14, when the pen is mounted (A), the menu area 410displays each menu 411, which is as large as the size that issufficiently selectable by a finger. In this state, when the pen isdetached (B), the controller 130 reduces the size of the menu area 410,and enlarges the size of the content display area 420. In other words,when the pen 200 is used, each menu 411 may be easily selected eventhough the size of the menu area 410 is reduced. In addition, as thesize of the content display area 420 is enlarged, a user may view apre-view image of each content 421 more easily. The ratio for adjustingthe size of the menu area 410 and the content display area 420 may beset in various ways according to the display size and ratio of the userterminal device 100.

If a program which allows a user to draw using the pen 200 is executed,the controller 130 may display a screen which is divided into a menuarea and a canvas area. In this case, when the pen is detached, thescreen may be resized such that the menu area is reduced and the canvasarea is enlarged just like the above case.

FIG. 15 is a view provided to explain another controlling method forchanging a layout of an application screen. According to FIG. 15, when amusic content play program is executed, the controller 130 displays theexecution screen 1510, 1520. If the program is executed while the pen200 is mounted (A) or if the pen 200 is mounted after the program isexecuted, the controller 130 displays the first execution screen 1510according to the first layout. The first execution screen 1510 maydisplay information on a reproduced content 1511, a control menu 1512 inthe size which is touchable by a finger, and a menu 1513 for checking alist of music contents which can be reproduced.

Moreover, if the program is executed while the pen is detached (B) or ifthe pen is detached after the program is executed, the controller 130displays the second execution screen 1520 according to the secondlayout. The second execution screen 1520 may display not onlyinformation on reproduced contents and a menu for checking a list ofmusic contents which can be reproduced 1550, but also a circular object1540.

The circular object 1540 may include various buttons 1541, 1542 whichcan be selectable by touch, a volume menu 1543, a reproduction stateadjustment menu 1544 which are adjustable through touch-and-drag by thepen 200, a reproduction state display area 1545, etc. A user may touchand drag the area where the volume menu 1543 is displayed in a clockwisedirection using the pen 200 to increase the volume, or may touch anddrag the area in a counterclockwise direction to decrease the volume. Inaddition, a user may touch and drag the area where the reproductionstate adjustment menu 1544 is displayed in a clockwise direction usingthe pen 200 to put forward a reproduction point and may touch and dragthe area in counterclockwise direction to put backward a reproductionpoint.

As illustrated in FIG. 15, if a circular object is provided, a user mayperform various interaction operations with a gesture of drawing acircle on the border of the circular object using the pen 200.

The circular object of FIG. 15 may be displayed in the form of disk. Inother words, a user may act like a deejay by grapping and scratching thedisk using the pen 200.

In FIG. 15, a circular object is illustrated, but various types of userinterfaces (UIs) which can be manipulated easily by the pen 200 may beprovided when the pen 200 is detached. Further, according to anotherexemplary embodiment, a user may touch an indicator area using the pen200, and control the operation of the user terminal device 100. Aninteraction operation of touching and controlling an indicator area willbe described in detail.

<Example of Controlling Indicator Area>

FIGS. 16 and 17 are views provided to explain a control operationaccording to an example of controlling an indicator area. According toFIG. 16, the controller 130 may display a screen 1600 including anindicator area 1610 where at least one indicator 1611 through 1618 aredisplayed.

Each indicator 1611 through 1618 is provided to inform an operationstate of the user terminal device 100. The indicators 1611 through 1618may include the indicator 1611 for informing an absent call, theindicator 1612 for informing a newly-received message or messenger, theindicator 1613 for informing a program update, the indicator 1614 forinforming a current communication method, the indicator 1615 forinforming a communication state, the indicator 1616 for informing aresidual battery, the indicator 1617 for informing whether an alarm isset, and the indicator 1618 for informing the time. The size, type andshape of the indicators are only an example, and the indicator area 1610may be displayed in other shapes. For example, an indicator forinforming a date, an indicator for informing a weather condition, anindicator for informing a predetermined schedule, an indicator forinforming whether an operation mode of the user terminal device is apower-saving mode or a general mode, an indicator for informing whethera current mode is a vibration mode or a bell-sound mode, an indicatorfor informing whether a Bluetooth is used, etc., may also be displayedin the indicator area 1610 depending on an exemplary embodiment.

The indicator area 1610 may be displayed all the time even if the typeof screen 1600 is changed, and the indicator area 1610 is generallydisplayed in a smaller size in comparison with other areas. Accordingly,it is not easy to touch the indicator area 1610 with a finger. However,the pen 200 which has a sharp end may directly touch the indicators 1611through 1618 which are displayed in the indicator area 1610.

The controller 130 operates in a general mode while the pen 200 ismounted on the user terminal device 100, and the controller 130 operatesin a pen use mode when the pen 200 is detached from the user terminaldevice 100. The mode may be changed automatically at the time whenwhether the pen is detached or not is detected.

When there is a touch to select an indicator in a general mode, thecontroller 130 disregards the touch manipulation. In other words, eachindicator 1611 through 1618 may not receive a user selection in ageneral mode. Further, when an indicator in the indicator area istouched by the pen 200 in a pen use mode, the controller 130 performs acontrol operation corresponding to the touched indicator.

For example, when an indicator for informing a current mode set in thecurrent user terminal device 100 is selected by the pen 200 from amongindicators, the controller 130 changes the shape of the indicator to ashape corresponding to another mode, and changes the current mode of theuser terminal device 100 to another mode.

FIG. 16 illustrates a state where the indicator 1614 for informing thatthe current communication method of a user terminal device is Long TermEvolution (LTE) is touched by the pen 200. In this case, the controller130 changes the current communication method to another communicationmethod (e.g., WiFi), and changes the indicator 1614 to the indicator1619 which corresponds to WiFi. In this state, if the indicator 1619 isselected again, the controller changes the indicator 1619 back to theindicator 1614, and also restores the communication method to theoriginal communication method. In other words, a user may directlycontrol various operations of the user terminal device 100 on theindicator area 1610 using the pen 200.

In addition to the communication mode, a vibration mode and a bell-soundmode may also be toggled and changed in the same manner.

Further, if a user performs a hovering with respect to variousindicators, the controller 130 may perform various control operationsaccording to the type of the indicators.

FIG. 17 is a view provided to explain a control operation when there isa hovering with respect to a message indicator. As illustrated in FIG.17, if there is a newly received message which is not yet checked by auser, the controller 130 displays the message indicator 1612. If theuser performs a hovering of approaching the pen 200 to the messageindicator 1612, the controller 130 displays a pop-up screen 1710corresponding to the message indicator 1612. The pop-up screen 1710 maydisplay at least part of the content of the new message.

In this state, if the user touches the corresponding message indicator1612 with the pen 200, the controller 130 displays a execution screen1720 of a message program. The user may check the entire message throughthe execution screen 1720, or may input a reply.

In addition, the controller 130 may perform various control operationsaccording to a user hovering or the type of touched indicator.

For example, if a user performs a hovering above the alarm indicator1617, the controller may display a pop-up screen including at least partof setting information regarding an alarm, such as information on thenext alarm time, information on alarm sound, etc. Alternatively, if auser performs a hovering above the clock indicator 1618, the controller130 may display a pop-up screen including various information regardingtime and date, such as time information, weather information,lunar-calendar date information, etc., of a predetermined country. Inaddition, if a user performs a hovering above the absent call indicator1611, the controller 130 may display a pop-up screen including thetelephone number of a absent call, and if a user performs a hoveringabove the notification indicator 1613, the controller 130 may display apop-up screen including a notification. Further, if a user performs ahovering above the communication quality indicator 1615 and the batteryindicator 1616, the controller 130 may provide information regarding acommunication state, a residual battery, etc. In addition, although notillustrated in FIGS. 16 and 17, when an indicator such as an earphone isdisplayed, the controller 130 may display a music control panel in apop-up screen when there is a hovering above the indicator.

The above exemplary embodiments describe various examples of changingthe layout of a screen and using the screen when the pen is detachedwhile the screen is displayed.

According to another exemplary embodiment, when there is a usermanipulation, a screen of different layouts may be provided depending onthe type of manipulation. In particular, the layout of a quick panelscreen which is displayed according to a simple user manipulation in thehome screen may be displayed differently according to the type ofmanipulation. Various control methods using the quick panel screen willbe described in detail.

<Example of Changing Quick Panel Screen>

FIG. 18 is a flowchart provided to explain a control method according toanother exemplary embodiment. According to FIG. 18, a user terminaldevice may monitor whether there is a general touch manipulation ofdragging from the border of the screen towards the center of the screenor there is a pen manipulation (S1820, S1840) while the screen isdisplayed (S1810).

The general touch manipulation refers to a manipulation with otherinputters than the pen 200, such as a body part like a finger, and thepen manipulation refers to a manipulation which uses the pen 200.

If it is determined that a user inputs a dragging from the border of thescreen towards the center of the screen using his or her finger (S1820),the controller 130 displays a first quick panel screen (S1830). Thequick panel screen refers to a screen which is provided to allow a userto set an operation mode or main functions of the user terminal devicequickly. The quick panel screen may also be referred to as a controlscreen, a setting screen, a mode setting screen, a notification screen,etc.

If it is determined that a user performs dragging using a pen (S1840),the controller 130 displays a second quick panel screen which has adifferent layout from the first quick panel screen (S1850).

In other words, if there is a dragging using a pen, it is highly likelythat there is a pen manipulation later. Thus, the controller 130 maydisplay the second quick panel screen which allows minute controloperations using the pen 200.

FIG. 19 is a view provided to explain a control operation according tothe exemplary embodiment of FIG. 18. According to FIG. 19, when there isa dragging from the border of the indicator area towards the center ofthe screen, i.e., in the lower direction while an arbitrary screen 1900is displayed, a hidden screen is scrolled and comes down according tothe dragging point.

The controller 130 determines whether the input of dragging is a pen ora user body part based on the detection result by the detector 120.

Accordingly, if it is determined that the dragging is a general touchmanipulation using a user body part, the controller 130 displays thefirst quick panel screen 1910. The first quick panel screen 1910 maydisplay a plurality of tap areas 1911 and various notification items1912. In FIG. 19, the notification items 1912 are illustrated in theform of bar, but it is only an example.

If it is determined that the dragging is a pen manipulation using thepen 200, the controller 130 displays the second quick panel screen 1920.The second quick panel screen 1920 may also display a plurality of tapareas 1921 and various notification items 1922.

Comparing the first quick panel screen 1910 with the second quick panelscreen 1920, it can be seen that the size of the taps 1921 and thenotification items 1922 displayed on the second quick panel screen 1920is smaller than that of the first quick panel screen 1910. Accordingly,the second quick panel screen 1920 may display more taps 1921 andnotification items 1922 than the first quick panel screen 1910.

FIG. 19 illustrates that the taps 1921 and the notification items 1922of the second quick panel screen 1920 are the same as those of the firstquick panel screen 1910, i.e., the taps 1911 and the notification items1912, and include other taps and notification items additionally, butthis is only an example. In other words, the taps 1921 and thenotification items 1922 of the second quick panel screen 1920 may bedifferent from those of the first quick panel screen 1910, i.e., thetaps 1911 and the notification items 1912. In addition, the layout ofthe quick panel screen may be changed. For example, the first quickpanel screen may display the notification items 1912 in the form of bar,and the second quick panel screen may display each notification item1922 in the form of checkerboard. A user may control the operations ofthe user terminal device 100 by inputting more minute touch and hoveringusing the pen 200 on the second quick panel screen 1920. Hereinafter,examples of various control operations using the pen 200 will bedescribed.

According to FIG. 20, a user may draw a predetermined pattern on thenotification item 1922 using the pen 200 while the second quick panelscreen is displayed. FIG. 20 illustrates that a user draws a pattern ofstroking downward straight. In this case, the controller 130 may deletethe entire notification items 1922 displayed on the second quick panelscreen 1920 all together. FIG. 20 illustrates the pattern of drawingdownward, but the type of patterns may be set in various ways. Forinstance, the controller 130 may perform the above-described deletingoperation when a user draws the patterns of X or O. Accordingly, theuser may delete a notification item directly without searching for themenu to delete the notification item.

FIG. 20 illustrates the operation of deleting the entire notificationitems all together, but a user may delete only a part of thenotification items by drawing a predetermined pattern such as pattern Xon at least one notification.

FIG. 21 is a view provided to explain another control operation using apen. According to FIG. 21, a user may take notes on a notification itemusing the pen while the second quick panel screen 1920 is displayed.When there is a note-taking by a pen manipulation on one notificationitem 2100, the controller 130 may perform a control operationcorresponding to the type of the notification item 2100 and the contentsof the note-taking automatically.

FIG. 21 illustrates a case where the text, “KIM”, is written on anotification item 2100 which is related to Internet. The controller 130executes a web browser matched with the notification item 2100 andaccess a predetermined web site. Subsequently, the controller 130searches for a keyword including “KIM” from the corresponding web site,and displays the search result directly on a screen 2120.

FIG. 22 illustrates a state where the text, “KIM”, is written on anotification item 2200 which is related to a messenger program. In thiscase, the controller 130 executes the messenger program automatically,and displays a messenger screen 2220 including messages exchanged withthe person called “KIM”.

In addition, if a user draws a specific pattern on a notification item(not shown) for informing reception of a mail, the controller 130 maydelete, store securely, or register as favorites the item according tothe type of the pattern.

FIGS. 23 to 25 are views illustrating a control method using a tap areaon the second quick panel screen.

According to FIG. 23, a user may write an arbitrary number on a tap 2300of lighting time using the pen 200. The controller 130 recognizes thenumber and changes a setting parameter of the corresponding tap 2300 toa value corresponding to the recognized number. As illustrated in FIG.23, if a user inputs the number “2” on the lighting time tap 2300, thecontroller 130 sets the time for providing light which brightens thescreen of the user terminal device 100 to 2 minutes. Subsequently, thecontroller 130 displays a pop-up screen 2310 for informing the user thatthe lighting time is set to 2 minutes. The pop-up screen 2310 may berealized as an on screen display (OSD) which is overlapped with theprevious screen, i.e., the second quick panel screen.

According to FIG. 24, a user may draw a specific pattern using the pen200. FIG. 24 illustrates a drawing of a spiral pattern in a clockwisedirection on a brightness tap 2400. In this case, the controller 130 mayincrease the brightness of a screen by increasing the setting parametervalue of the brightness tap 2400. Further, if a user draws a spiralpattern in a counterclockwise direction, the controller 130 may reducethe setting parameter value of the brightness tap 2400. The controller130 may display a pop-up screen 2410 which displays brightnessadjustment. In FIG. 24, the spiral pattern is illustrated as an example,but various types of patterns may be used. For example, drawing of astraight line may also change the setting parameter value according toits direction.

Further, a user may perform a hovering with respect to a tap area or anotification item on the second quick panel screen in order to checkinformation regarding the tap or the notification item.

FIG. 25 is a view provided to explain a control operation when there isa hovering with respect to a tap area. According to FIG. 25, when a userperforms a hovering over a WiFi tap 2500, the controller 130 may displaya pop-up screen 2510 including information regarding acurrently-connected WiFi Access Point (AP) or information regarding WiFiintensity.

As described above, a user may use the operations of the user terminaldevice in various ways using the pen 200. The controller 130 mayrecognize a user manipulation of the pen 200 using the above-describedpen recognition module, and perform a control operation intended by theuser by executing various programs accordingly.

In particular, the controller 130 may provide different layoutsdepending on whether a pen is attached or detached. The controller 130may change the layout by selectively executing a program or data whichhas two versions, or by adjusting the resolution of the execution screenof a single program. Accordingly, different usage environments areprovided depending on whether it is a touch mode or a pen use mode.

As described above, a user may draw an object directly on the homescreen using the pen 200 so as to design his or her own home screenaccording to personal preference. As the pen 200 allows a fine touch, auser may directly draw a small object on the home screen and match theobject with a control operation. Accordingly, it is possible to displaya screen which is optimized for the pen use mode.

FIG. 26 is a view illustrating an example of a home screen which isdesigned directly by a user. According to FIG. 26, a home screen 2600 ofthe user terminal device 100 displays various objects 2610 through 2690.Each of the objects 2610 through 2690 is generated in response to a userdirectly drawing on the home screen 2600 using an inputters such as hisor her finger or the pen 200. Accordingly, the display attributes ofeach object 2610 through 2690, such as size, shape, location, color,etc., are not fixed, but they may be displayed in various ways dependingon each user. Each object may be described as an individual object, ormay be described to include a plurality of objects 2621 through 2624,similar to the object 2620.

A user may match an object which he or she draws with a desired controloperation. If a user matches a control operation with each object, thecontroller 130 stores the matching information in the storage 140 or inother storage device.

Subsequently, when the displayed object is selected, the controller 130performs the control operation matched with the object based on thematching information. If the object 2620 including a plurality ofobjects 2621 through 2624 is selected, the controller 130 may perform aplurality of control operations matched with each object 2621 through2624.

It is not necessary to match a control operation to all of the objects.Some of the objects may be used only as drawings for visualsatisfaction. In FIG. 26, every object is drawn directly by a user, butsome of the objects may be provided by the user terminal device 100 as adefault.

Further, a user may select a control operation before drawing an object.In this case, the controller 130 may provide an icon corresponding tothe selected control operation in advance, similar to a water mark. Theuser may draw an object following the water mark. Alternatively, thecontroller 130 may display various images in relation with a selectedcontrol operation in advance so that the user may use one of the imagesas a reference.

FIG. 27 is a view illustrating that different home screens are provideddepending on whether a pen is attached or detached. If the pen ismounted (A), the controller 130 may display a default screen 2700 whichis provided by a manufacturer of the user terminal device 100 as adefault. In this state, if the pen is detached (B), the controller 130may display the user creative screen 2600 which is designed directly bya user. The storage 140 may store home screen data to produce thedefault screen 2700 or home screen data to produce the user creativescreen 2600. The controller 130 may use such home screen data todifferentiate not only the layout of screen but also the screen itselfdepending on whether the pen is attached or detached.

When the user creative screen 2600 is displayed after the pen isdetached, a user may redesign the user creative screen 2600 by adding ormodifying various objects using the pen 200.

In addition, it is possible to change the shape of only a specificwidget depending on whether the pen is attached or detached.Hereinafter, the examples of changing the shape of widget will bedescribed in detail.

<Change of Widget Depending on Whether Pen is Detached>

FIG. 28 is a view provided to explain a control method of a userterminal device according to another exemplary embodiment. According toFIG. 28, the user terminal device 100 displays a screen 2800 includingvarious widgets 2810, 2820 and an icon 2830. FIG. 28 illustrates thescreen 2800 including the first widget 2810 for displaying weather,time, date, etc., the second widget 2820 for adjusting a contentreproduction state, etc. The first widget 2810 is displayed all the timeregardless of whether the pen is mounted (A) or detached (B). The shapeof the second widget 2820 changes depending on whether the pen isdetached or not.

In other words, when the pen is mounted (A), the controller 130 displayseach button of the second widget 2820 in the size and shape which can beeasily touched by a user finger. In this state, if the user detaches thepen 200 from the user terminal device 100, the controller 130 changesthe mode to a pen use mode. Accordingly, the second widget 2820 ischanged to a circular object 2840.

According to FIG. 28, the circular object 2840 may include variousbuttons 2841, 2842 which are selectable by a touch, a volume menu 2843or a reproduction state adjustment menu 2844 which may be adjustable bya touch-and-drag of the pen 200, a reproduction state display area 2845,etc. As described above with reference to FIG. 15, a user may adjustvolume by touching and dragging an area where the volume menu 2843 isdisplayed using the pen 200, or adjust a reproduction time by touchingand dragging an area where the reproduction state adjustment menu 2844is displayed using the pen 200. The configuration and operation of thecircular object 2840 of FIG. 28 have already described above withreference to FIG. 15, so further description will not be provided.

FIG. 28 illustrates that only the shape of one widget 2840 is changed,but the display location, size, number, etc., of the widget may also bechanged. In addition, the shape, size, location, etc., of another widget2810 or the icon 2830 may also be changed depending on circumstances.

FIG. 29 is a view provided to explain another example of changing awidget according to an interaction mode. According to FIG. 29, when thepen is mounted (A), the display 110 displays a screen 2900 including awidget 2910.

The controller 130 operates in a general mode when the pen 200 ismounted on the user terminal device 100, and operates in a pen use modewhen the pen 200 is detached from the user terminal device 100.

When there is a touch on a widget in a general mode, the controller 130executes a program corresponding to the widget 2910. Accordingly, whenan execution screen of the corresponding program is displayed, a usermay change displayed contents of the widget 2910 by inputting variousdata related to the widget 2910 using the execution screen.

On the other hand, when a pen manipulation is input on the widget 2910in a pen use mode, the controller 130 adjusts the setting data of thewidget 2910 according to the pen manipulation, and changes the displayedcontents of the widget 2910 according to the changed setting data.

FIG. 29 illustrates a case where a widget is an alarm widget 2910including an analog clock 2940 which includes an hour hand 2941 and aminute hand 2942. A digital clock 2920, weekday information 2930, etc.,may also be displayed in the alarm widget 2910.

When there is a touch on the alarm widget 2910 in a general mode, thecontroller 130 executes an alarm program and displays an executionscreen, i.e., an alarm setting screen. A user may input various datasuch as a desired time, alarm frequency, alarm date, and bell sound onthe alarm setting screen.

In a pen use mode, the controller 130 sets an alarm according to a penmanipulation on the alarm widget 2910, and changes displayed contents ofthe widget according to the set alarm.

In particular, the controller 130 may display only part of numbers (12,3, 6, 9) indicating time in the analog clock 2940 of the widget 2910.Then, when there is a pen hovering (B), the entire numbers (1 through12) may be displayed. In this state, a user may touch the hour hand 2941and the minute hand 2942 using the pen 200, respectively, and drag 2950them to a desired location.

When a pen manipulation to move at least one of the hour hand 2941 andthe minute hand 2942 is input on the analog clock 2940, the controller130 sets an alarm according to the time changed by the pen manipulation.Subsequently, the controller 130 may move at least one of the hour handand the minute hand to the time when the alarm is set. FIG. 29illustrates a state where the location of the hour hand 2941 is changedfrom 11 to 5, and the minute hand 2942 remains at 12. Accordingly, thealarm is set to 5 o'clock. When the alarm setting time is changed, thetime of the analog clock 2940 and the digital clock 2920 is changedautomatically.

A user may directly select a date where the user wishes to set off analarm by touching the weekday information 2930 with the pen 200. Theselected date may be displayed in a different color from the otherdates. In addition, although not illustrated in FIG. 29, an area forselecting the morning and the afternoon, an area for selecting on/off ofalarm, an area for selecting a bell sound may also be included in thewidget 2910.

FIG. 30 is a view provided to explain another method for selecting analarm in the alarm widget 2910. According to FIG. 30, a user may take anote regarding time information 2960 on the digital clock 2920 using thepen 200. FIG. 30 illustrates a case where ‘1’ is written in the hourarea and ‘27’ is written in the minute area. The controller 130recognizes the notes and sets an alarm setting time according to therecognition result. In addition, the controller 130 changes thedisplayed contents of the digital clock 2920 and the analog clock 2910according to the set time.

As described above, an icon and a widget may be displayed differentlydepending on whether it is a general mode and a pen use mode, and theirutilizing method may also be changed according to mode.

As described above, according to various exemplary embodiments, a usermay select a desired control operation conveniently on the screen whereobjects are displayed, using various manipulation methods. Accordingly,user convenience in executing and using various programs can be improvedsignificantly. Each of the above exemplary embodiments may be realizedand applied individually, but they may also be combined and applied. Inother words, a plurality of objects may be grouped in one user terminaldevice 100 and a plurality of control operations may be performed.Alternatively, a plurality of control operations may be matched with oneof the objects, and various control operations may be selectivelyperformed according to the direction or location of a user manipulationof selecting an object or the contents input by the user.

As described above, a user terminal device may be controlled in variousways according to a user body part or a pen. The example of such acontrol operation may combined and realized in various ways depending onthe type of the user terminal device 100 and the characteristics ofprograms installed therein.

For example, as described above, a user terminal device may be realizedas various types of apparatuses such as mobile phone, tablet PC, laptopPC, PDA, MP3 player, electronic album device, TV, PC, kiosk, etc. If auser terminal device is realized as an apparatus having variousfunctions like a communication function, a content reproductionfunction, etc., such as mobile phone, tablet PC, etc., the user terminaldevice may further include other elements to perform such functions.

FIG. 31 is a block diagram illustrating an example of a user terminaldevice including various elements in comprehensive manner.

According to FIG. 31, the user terminal device 100 may include thedisplay 110, the detector 120, the controller 130, the storage 140, acommunicator 150, a Global Positioning System (GPS) chip 161, a videoprocessor 162, an audio processor 163, a button 164, a microphone 165, acamera 166, a speaker 167, and a motion detector 168.

The display 110 and the detector 120 may be realized in the form oftouch screen as described above, display various screens, and detect auser manipulation on the screen.

The controller 130 controls overall operations of the user terminaldevice using various programs and data stored in the storage 140. Inparticular, the controller 130 may detect whether the pen 200 isattached or detached, and display screens of various layouts accordingto the detection result.

As the display 110, the detector 120, and the controller 130 havealready been described above, further description will not be provided.

The communicator 150 is an element which performs communication withvarious types of external apparatuses according to various types ofcommunication methods. The communicator 150 includes a WiFi chip 151, aBluetooth 152, a wireless communication chip 153, and an NFC chip 154.

The WiFi chip 151 and the Bluetooth chip 152 perform communicationaccording to a WiFi method and a Bluetooth method, respectively. In thecase of the WiFi chip 151 or the Bluetooth chip 152, various connectioninformation such as SSID and a session key may be transmitted/receivedfirst for communication connection. Then, various information may betransmitted/received using the same. The wireless communication chip 153represents a chip which performs communication according to variouscommunication standards such as IEEE, Zigbee, 3^(rd) Generation (3G),3^(rd) Generation Partnership Project (3GPP), Long Term Evolution (LTE),etc. The NFC chip 154 represents a chip which operates according to anNFC method which uses 13.56 MHz band among various RF-ID frequency bandssuch as 135 kHz, 13.56 MHz, 433 MHz, 860 through 960 MHz, 2.45 GHz, etc.

The controller 130 performs communication with various externalapparatuses using the communicator 150. For example, the controller 130may download an application, install the application in the storage 140,and display an object corresponding to the application. In this case,the corresponding application may be matched with a previously-usedobject.

In particular, the communicator 150 may perform communication with anexternal server. The server may be a server for providing a sharingservice which shares a home screen including objects which are drawndirectly by another user. In this case, when a command to share a homescreen is input in the user terminal device 100, home screen data whichconstitutes the home screen produced by a user may be uploaded to theserver using the communicator 150. Alternatively, new home screen datawhich is generated by another user may be transmitted from the server.When the new home screen data is downloaded, and if a user inputs a homescreen change command to change the home screen by selecting the newhome screen data, the controller 130 may display the new home screenbased on the new home screen data. In addition, the communicator 150 mayshare home screen data by performing communication directly with anexternal terminal device of another user instead of the server.

The GPS chip 161 is an element to calculate the current location of theuser terminal device 100 by receiving a GPS signal from a GPS satellite.When a navigation program is used or when it is necessary to figure outthe current location of a user, the controller 130 may calculate theuser location using the GPS chip 161.

The video processor 162 is an element to process contents receivedthrough the communicator 150 or video data included in the contentsstored in the storage 140. The video processor 162 may perform variousimage processing with respect to video data such as decoding, scaling,noise filtering, frame rate conversion, resolution conversion, etc.

The audio processor 163 is an element to process contents receivedthrough the communicator 150 or audio data included in the contentsstored in the storage 140. The audio processor 163 may perform variousprocessing with respect to audio data such as decoding, amplifying,noise filtering, etc.

When a reproduction program with respect to multimedia contents isexecuted, the controller 130 drives the video processor 162 and theaudio processor 163 to reproduce the corresponding contents. The display110 may display an image frame which is generated by the video processor162.

In addition, the speaker 167 outputs audio data generated by the audioprocessor 163.

The button 164 may be a button in various forms such as a mechanicbutton, a touch pad, a wheel, etc., which is formed on a certain area ofthe user terminal device 100 such as the front, side, or back of theexterior of the main body.

The microphone 165 is an element to receive a user voice or other soundsand convert the same into audio data. The controller 130 may use a uservoice input through the microphone 165 during a call, or may convert theuser voice into audio data and store the same in the storage 140.

The camera 166 is an element to photograph a still image or a videounder the control of a user. A plurality of cameras 166 such as a frontcamera and a rear camera may be provided.

When the camera 166 and the microphone 165 are provided, the controller130 may perform a control operation according to a user voice inputthrough the microphone 165 and a user motion recognized by the camera166. In other words, the user terminal device 100 may operate in amotion control mode or in a voice control mode. When the user terminaldevice 100 operates in the motion control mode, the controller 130activates the camera 166 to photograph a user, trace a change in theuser motion, and perform a corresponding control operation. When theuser terminal device 100 operates in the voice control mode, thecontroller 130 may operate in the voice recognition mode where a uservoice input through the microphone 165 is analyzed and a controloperation is performed according to the analyzed user voice.

In the user terminal device 100 where the motion control mode or thevoice control mode is supported, a voice recognition technology and amotion recognition technology may be used according to theabove-described various exemplary embodiments. For example, if a usertakes a motion of selecting an object displayed on the home screen or ifa user utters a voice command corresponding to the object, it isdetermined that the corresponding object is selected, and a controloperation matched with the object may be performed.

The motion detector 168 is an element to detect a motion of the mainbody of the user terminal device 100. In other words, the user terminaldevice 100 may rotate or tilt in various directions. The motion detector168 may detect characteristics of motion, such as rotation direction,angle, tilt, etc., using at least one of various sensors like ageomagnetic sensor, a gyro sensor, an acceleration sensor, etc.

Although not illustrated in FIG. 31, depending on exemplary embodiments,the user terminal device 100 may further include a USB port where a USBconnector can be connected, various external input ports for connectionwith various external terminals such as a headset, a mouse, LAN, etc.,and a DMB chip which receives and processes a Digital MultimediaBroadcasting (DMB) signal.

Meanwhile, the operation of the controller 130 may be performed byexecuting various software stored in the user terminal device 100.

FIG. 32 is a view illustrating an example of a software structure whichis used by the user terminal device 100. The software of FIG. 32 may bestored in the storage 140, but is not limited thereto. The software ofFIG. 32 may be stored in various types of storage device which are usedin the user terminal device 100. According to FIG. 32, the user terminaldevice 100 may store software including an Operating System (OS) 141, akernel 142, middleware 143, an application 144, etc.

The OS 141 controls overall operations and manages functions ofhardware. In other words, the OS 141 is responsible for basic functionsof managing hardware, memory, security, etc.

The kernel 142 serves as a path which transmits various signals detectedby the detector 120, etc., to the middleware 143.

The middleware 143 includes various software modules which control theoperations of the user terminal device 100. According to FIG. 32, themiddleware 143 includes an X11 module 143-1, an APP manager 143-2, aconnection manager 143-3, a security module 143-4, a system manager143-5, a multimedia framework 143-6, a UI framework 143-7, a windowmanage 143-8, and a note-taking recognition module 143-9.

The X11 module 143-1 is a module which receives various event signalsfrom various hardware of the user terminal device 100. The event mayinclude various events such as an event where a user manipulation isdetected, an event where a system alarm occurs, a event where a specificprogram is executed or terminated, etc.

The APP manager 143-2 is a module which manages the execution state ofvarious applications installed in the storage 140. When an event wherean application execution command is input is detected from the X11module 143-2, the APP manager 143-2 calls and executes an applicationcorresponding to the event. In other words, when an event where at leastone object is selected on the home screen is detected, the APP manager143-2 calls and executes an application corresponding to the object.

The connection manager 143-3 is a module which supports wired orwireless network connection. The connection manager 143-3 may includevarious specific modules such as a DNET module, an UPnP module, etc.

The security module 143-4 is a module which supports certification ofhardware, permission of request, security storage, and so on.

The system manager 143-5 monitors the state of each element of the userterminal device 100, and provides the monitoring result to othermodules. For example, if a battery is running short, there is an error,or communication connection is cut off, the system manager 143-5 mayoutput an alarm message or an alarm sound by providing the monitoringresult to the UI framework 143-7.

The multimedia framework 143-6 is a module which reproduces a multimediacontent stored in the user terminal device 100, or provided by anexternal source. The multimedia framework 143-6 may include a playermodule, a camcorder module, a sound processing module, and so on.Accordingly, the multimedia framework 143-6 may perform the operation ofreproducing various multimedia contents and generating and reproducingscreen and sound.

The UI framework 143-7 is a module to provide various UIs. The UIframework 143-7 may include an image compositor module for configuringvarious objects, a coordinates compositor module for calculating acoordinates where an object is to be displayed, a rendering module forrendering the configured object on the calculated coordinates, a 2D/3Dtoolkit for providing a tool for configuring a UI in the 2D or 3D form,etc.

The window manager 143-8 may detect a touch event using a user body partor pen or other input events. When such an event is detected, the windowmanager 143-8 transmits an event signal to the UI framework 142-7 sothat an operation corresponding to the event is performed.

The note-taking recognition module 143-9 is a module to parse andrecognize a trace that a user draws on the surface of the display 110.The note-taking recognition module 143-9 receives a coordinates value ofpoints touched by the pen recognition panel 121 and stores thecoordinates values of the touched points as a stroke. In addition, thenote-taking recognition module 143-9 may generate a stroke array usingthe stroke. Subsequently, the note-taking recognition module 143-9 mayrecognize a note-taking by comparing a pre-stored note-taking librarywith the generated stroke array. The recognized note-taking is providedto the application manager 143-2, and the application manager 143-2provides the note-taking to an application corresponding to the objectwhere the user note-taking occurs. Accordingly, as described above, theapplication may perform a control operation based on the recognizednote-taking.

Meanwhile, if a user touches or draws the surface of the display 110using various inputters such as a user finger or pen, the UI framework143-7 may draw various graphic lines according to the user manipulation.In particular, if a user draws a picture on the screen using the pen200, the controller 130 draws a graphic line on each point where the pen200 passes through according to the execution of the UI framework 143-7.The thickness of the line may vary according to the pressure of the pen200 on the screen or the size of the screen touched by the pen 200. Thecolor and shape of the line may also vary according to a user setting.

In addition, when an event of detaching or attaching a pen occurs, theUI framework 143-7 performs the operation of changing a layout of the UIscreen which is currently displayed.

The application module 144 includes applications 144-1 through 144-n tosupport various functions. For example, program modules to providevarious services such as a navigation program module, a game module, anelectronic book module, a calendar module, an alarm management module,etc., may be included. The applications may be installed as a default,or may be installed and used arbitrarily during the usage process.

The software structure of FIG. 32 is only an example, so the softwarestructure is not limited thereto. Accordingly, part of the softwarestructure may be omitted, changed, or added, if necessary. For example,the storage 140 may further include various programs like a sensingmodule to analyze signals sensed by various sensors, a messengerprogram, a messaging module such as Short Message Service (SMS) &Multimedia Message Service (MMS) program, an e-mail program, etc., acall info aggregator program module, a VoIP module, a web browsermodule, etc.

As described above, according to various exemplary embodiments, theapplicability of a user terminal device may be improved significantly.In particular, where interaction of accurate pen manipulation is allowedin a screen, a user may conveniently use a pen in various ways.

The method of controlling a user terminal device according to the abovevarious exemplary embodiments may be stored in a non-transitory readablemedium. The non-transitory readable medium may be mounted and used invarious apparatuses.

For example, a program code for displaying a screen, when a pen mountedon the user terminal device is detached, changing the layout of thescreen to a layout corresponding to a pen use mode, when the pen ismounted on the user terminal device again, restoring the layout of thescreen to its original state may be stored and provided in anon-transitory readable medium.

The non-transitory recordable medium refers to a medium which may storedata semi-permanently rather than storing data for a short time such asa register, a cache, and a memory and may be readable by an apparatus.Specifically, the above-mentioned various applications or programs maybe stored in a non-temporal recordable medium such as CD, DVD, harddisk, Blu-ray disk, USB, memory card, and ROM and provided therein.

The foregoing embodiments and advantages are merely exemplary and arenot to be construed as limiting the exemplary embodiments. The presentteaching can be readily applied to other types of apparatuses. Also, thedescription of the exemplary embodiments is intended to be illustrative,and not to limit the scope of the claims. Many alternatives,modifications, and variations will be apparent to those skilled in theart.

What is claimed is:
 1. A user terminal device comprising: a display; adetector; and a controller configured to: control the detector toidentify a touch of a user and an input of a pen, the detector being ahardware element, control the display to display a screen including afirst area and a second area, wherein the first area includes aplurality of objects and the second area includes at least oneindicator, control the display to display the plurality of objects onthe first area by arranging the plurality of objects on the first areaaccording to a first layout in a first mode, and control the display todisplay the plurality of objects on the first area by re-arranging theplurality of objects on the first area according to a second layout in asecond mode, the second mode being a mode in which the pen is used,wherein the at least one indicator is selectable by the input of thepen, wherein the plurality of objects are selectable by the touch of theuser and the at least one indicator is not selectable by the touch ofthe user in the first mode, wherein the controller is configured todisplay a first quick panel screen based on a touch draggingmanipulation being identified from an edge area of the screen towards acenter area of the screen, wherein the controller is configured todisplay a second quick panel screen based on a pen dragging manipulationbeing identified from the edge area of the screen towards the centerarea of the screen, and wherein based on a pen manipulation in apredetermined pattern being identified on the second quick panel screen,the controller is configured to perform a control operation according tothe predetermined pattern.
 2. The user terminal device as claimed inclaim 1, wherein the controller is further configured to identifywhether the pen is removed from the user terminal device and arrange theplurality of objects according to the second layout.
 3. The userterminal device as claimed in claim 1, wherein the first layout and thesecond layout are different from each other in terms of displayattributes of the plurality of objects.
 4. The user terminal device asclaimed in claim 2, wherein the first area includes a menu areaconfigured to display at least one menu which is selectable by the penand a content display area configured to display at least one content,wherein the controller is configured to reduce a size of the menu areaand enlarge a size of the content display area to display the first areaaccording to the second layout based on the pen being removed from theuser terminal device, and wherein the controller is configured torestore the size of the menu area and the size of the content displayarea to the first area according to the first layout based on the penbeing mounted in the user terminal device.
 5. The user terminal deviceas claimed in claim 2, wherein the controller is configured to reduce asize of each object of the objects, and increase a number of totalobjects displayed on the first area.
 6. The user terminal device asclaimed in claim 1, wherein the controller is configured to change ashape of a selected indicator to a shape corresponding to another mode,and convert a current mode of the user terminal device to the anothermode based on an indicator corresponding to a current mode of the userterminal device being selected by the pen.
 7. The user terminal deviceas claimed in claim 1, wherein the first quick panel screen and thesecond quick panel screen are configured to display at least onerespective notification item, wherein the controller is configured toperform a control operation corresponding to a notification item and anote-taking automatically based on the note-taking being performed usingthe pen on the notification item displayed on the second quick panelscreen.
 8. A method for controlling a user terminal device comprising:displaying a screen which comprises a first area and a second area,wherein the first area includes a plurality of objects and the secondarea includes at least one indicator; displaying the plurality ofobjects on the first area by arranging the plurality of objects on thefirst area according to a first layout in a first mode, and displayingthe plurality of objects on the first area by re-arranging the pluralityof objects on the first area according to a second layout in a secondmode, the second mode being a mode in which a pen is used; and whereinthe at least one indicator is selectable by an input of the pen, whereinthe plurality of objects are selectable by a touch of a user and the atleast one indicator is not selectable by the touch of the user in thefirst mode, where the method further comprises: displaying a first quickpanel screen based on a touch dragging manipulation being identifiedfrom an edge area of the screen towards a center area of the screen;displaying a second quick panel screen based on a pen draggingmanipulation being identified from the edge area of the screen towardsthe center area of the screen; and based on a pen manipulation in apredetermined pattern being identified on the second quick panel screen,performing a control operation according to the predetermined pattern.9. The method as claimed in claim 8, wherein the first area includes amenu area configured to display at least one menu which is selectable bythe pen and a content display area configured to display at least onecontent, wherein the displaying the plurality of objects on the firstarea by re-arranging the plurality of objects on the first areacomprises reducing a size of the menu area and enlarging a size of thecontent display area based on the pen being removed from the userterminal device.
 10. The method as claimed in claim 8, wherein thedisplaying the plurality of objects on the first area by re-arrangingthe plurality of objects on the first area comprises reducing a size ofeach object of the objects, and increasing a number of total objectsdisplayed on the first area based on the pen being removed from the userterminal device.
 11. The method as claimed in claim 8, furthercomprising: changing a shape of a selected indicator to a shapecorresponding to another mode, and converting a current mode of the userterminal device to the another mode based on an indicator correspondingto the current mode of the user terminal device being selected by thepen.
 12. The method as claimed in claim 8, further comprising:performing a control operation corresponding to a notification item anda note-taking automatically based on the note-taking being performedusing the pen on the notification item displayed on the second quickpanel screen.
 13. A user terminal device capable of attaching anddetaching a pen, comprising: a display; a detector; and a controllerconfigured to: control the detector to identify a manipulation of auser, the detector being a hardware element; control the display todisplay a home screen including a first area and a second area, whereinthe first area is divided into a plurality of pages and is convertibleby a page and the second area includes at least one indicator; convertand display the first area by a page according to the manipulation in afirst mode, and reduce each size of the plurality of pages and displayentire pages within a single first area in a second mode, the secondmode being a mode in which the pen is used, wherein at least one objectis displayed on each page of the first area, wherein the at least oneindicator is selectable by an input of the pen, and wherein a pluralityof objects are selectable by a touch of the user and the at least oneindicator is not selectable by the touch of the user in the first mode.14. The user terminal device as claimed in claim 13, wherein thecontroller moves an object to another page according to a penmanipulation based on the pen manipulation for moving the objectincluded in one page to the another page being identified while theentire pages are displayed on the single first area.
 15. A user terminaldevice comprising: a display; a detector; and a controller configuredto: control the detector to identify a manipulation of a user regardinga screen, the detector being a hardware element; control the display todisplay a screen including a first area and a second area, wherein thefirst area includes a plurality of objects and the second area includesat least one indicator; control the display to display the plurality ofobjects on the first area by arranging the plurality of objects on thefirst area according to a first layout based on a pen being mounted onthe user terminal device, and control the display to display theplurality of objects on the first area by rearranging the plurality ofobjects on the first area according to a second layout corresponding toa pen use mode based on the pen being removed from the user terminaldevice, wherein the controller is configured to perform a controloperation corresponding to a selected object based on at least one ofthe plurality of objects being selected by the pen while the first area,which is changed according to the second layout, is being displayed, andwherein the controller is configured to rearrange the plurality ofobjects according to the first layout based on the pen being re-mountedon the user terminal device, wherein the at least one indicator isselectable by an input of the pen, wherein the plurality of objects areselectable by a touch of the user and the at least one indicator is notselectable by the touch of the user if the pen is mounted on the userterminal device, wherein the controller is configured to display a firstquick panel screen based on a touch dragging manipulation beingidentified from an edge area of the screen towards a center area of thescreen, wherein the controller is configured to display a second quickpanel screen based on a pen dragging manipulation being identified fromthe edge area of the screen towards the center area of the screen, andwherein based on a pen manipulation in a predetermined pattern beingidentified on the second quick panel screen, the controller isconfigured to perform a control operation according to the predeterminedpattern.